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Abstract 



OPERA is an experiment dedicated to the observation of into Vj oscillations in appearance mode using a pure beam (CNGS) 
0^ produced at CERN and detected at Gran Sasso. The experiment exploits a hybrid technology with emulsions and electronics 
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detectors |1]. The OPERA readout is performed through a triggerless, continuously running, distributed and highly available 
system. Its global architecture is based on Ethernet-capable smart sensors with microprocessing and network interface directly at 
the front-end stage. An unique interface board is used for the full detector reading out ADC-, TDC- or Controller-boards. All the 
readout channels are synchronized through a GPS-locked common bidirectional clock distribution system developped on purpose 
in a PCI format. It offers a second line to address all channels and the off-line synchronization with the CNGS to select the events. 
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1. Global data readout architecture 

OPERA is optimized for the r neutrino detection. Detector's 
target consists of lead-emulsions bricks walls followed by scin- 
tillator tracker planes (Target Tracker referred to as TT) to trig- 
ger and locate the events and it is complemented by two muon 
magnetic spectrometers (drift tubes referred to as HPT and RPC 
tracker). The overall data rate is dominated by cosmics and 
ambient radioactive background while the neutrino interactions 
are well localized in time in correlation with the CNGS beam 
spill. The synchronization with the CNGS beam spill is done 
off-line via the GPS. The detector remains sensitive during the 
inter-spill time and the DAQ system runs continuously in a trig- 
gerless mode. 

The data readout system has been designed to sort the data 
through Ethernet at the earliest stage of each sub-detector. 
The global DAQ is build like a standard Ethernet network 
whose nodes are the "Ethernet Controller Mezzanines" (ECM) 
plugged on Controller Boards (CB). These boards are designed 
to interface and control each sub-detector specific F/E electron- 
ics, to sort the data to the Event Building Workstation (EBWS), 
to handle monitoring and slow control from the Global Manager 
through the same Ethernet processors. 

Target Tracker The readout of the TT PMTs (Hamamatsu 64 
pixels H8804-modl) is performed via multi-channels front-end 
chips designed at LAL the signals of which are digitized and 
pre-processed on a custom motherboard developed at the IPNL. 
The F/E chips are self-triggering and allow to compensate the 
pixel-to-pixel gain spread of the MaPMT. A common thresh- 
old is set on a chip, which makes the OR of the 32 channels to 
provide a trigger output signal whenever one channel at least is 
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over threshold. 

Spectrometer Each spectrometer has two sensitive elements: 22 
RPC planes and 3 pairs of drift tubes stations located upstream, 
in the middle and downstream of a dipolar magnet. The elec- 
tronics of these 2 trackers is placed on the top of the experi- 
ment. It consists in VME-like boards collected in VME crates 
and power supply stations. The F/E electronics of one RPC 
plane is divided into 9 F/E boards readout serially by one Con- 
troller Board that receives the trigger signals (defined by the 
fast OR of 32 channels). Each drift tube station is readout by a 
TDC board which receives a common stop from the OR of two 
nearby RPC planes. 

Computing The EBWS consists of commercial PC's receiving 
data from the Ethernet network. The distributed client/server 
software is based on the CORBA (Common Object Request 
Broker Architecture) standard which is a well established Ob- 
ject Oriented application |2]. It offers a global framework 
which links the applications running on the sensors, on the 
EBWS and on the Manager. The on-line data storage is per- 
formed by a database cluster (Oracle 11.1 RAC) with two ded- 
icated servers. 

Network The global DAQ network is divided into 2 parts : the 
Ethernet network from the EBWS to each Ethernet Controller 
(used for data transfer but also for detector configuration, mon- 
itoring & slow control) and the clock distribution system from 
the Central Clock Unit synchronized on the GPS to each sensor 
(used for the synchronization of all the nodes local clocks) 

2. Ethernet Controller Mezzanine 

The aim of the ECM is to provide a common interface be- 
tween the F/E of the various sub-detectors and the overall DAQ 
system. The physical form of the mezzanine has been chosen 
for the ease of integration (see Figure [T]). The mezzanine can 
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Figure 1 : Ethernet Controller Module: schematics and physical view. 

be seen as an external component to be plugged on the Con- 
troller Boards and that can be (re-)configured to the specifica- 
tions of each sub-detectors fsl 0, Ed . The Ethernet controller is 
the core of the board. It includes a sequencer (FPGA from the 
Cyclone ALTERA family), an external FIFO and a micropro- 
cessor (ETRAX lOOLX from AXIS) with an Ethernet interface. 
The FPGA performs the sequencing of the readout (clocks, R/0 
registers, digital I/O), the local data pre-processing (zero sup- 
pression, event validation through external trigger on request), 
event time stamping (via a local fine counter at 100 MHz syn- 
chronized with the global distributed clock), data transfer to the 
external FIFO, interface with the Ethernet processor. The Eth- 
ernet processor is an ASIC based on the ETRAX lOOLX from 
AXIS. It is a 32-bit RISC CPU with Linux 2.4 operating sys- 
tem and Ethernet interface supporting data transfer rates up to 
200 Mbits/s 10]. The chip is embedded in a Multi Chip Module 
(MCM) which includes in a single chip the ETRAX lOOlx core 
plus 4 Mbytes of flash memory, 16 Mbytes of SDRAM and an 
Ethernet transceiver. The MCM gets the data in a time sched- 
uled job from the FIFO, performs local processing and transfers 
the data to the EBWS. 

3. Clock distribution system 

A global clock is mandatory to synchronize all the nodes 
of the distributed system. The general features of the system 
are close to Ref.|7], in particular the bi-directionnality of the 
system allows the control of the signal reception and the mea- 
surement of the propagation time with acknowledgement sig- 
nals. The clock distribution system starts from the GPS con- 
trol unit which synchronizes a 20MHz clock with the signal of 
the GPS and sends the clock -h encoded commands via an op- 
tical fibre. The signal is then converted into electrical format 
and distributed to the "clock master cards" through M-LVDS 
bus. Each of these cards deserialize the commands and the 
clock, and distribute both of them to the clock unit of each 
controller board through another M-LVDS bus. The PPS sent 
to each sensor is used to reset the local fine counters after cor- 
rection of the time propagation delay. Since the PPS transmis- 
sion time is recorded and locked with the GPS, the absolute 
event timestamp is assigned as the sum of that time and the 
value of the local lOOMHz clock running on the ECM. This 
timestamp accuracy allows monitoring the beam spill structure 
(Figure [2]) and to perform physics studies in parallel to the neu- 
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Figure 2: CNGS beam time structure seen in OPERA 

trino oscillations using particles ToF (cosmic rays physics, at- 
mospheric neutrinos etc). A crucial point to extract the beam 
events is the intercalibration between CERN and LNGS GPS 
systems. Both have comparable performance (jlOO ns accu- 
racy) and their single units are in both cases based on a GPS 
system -h Rb clock although build by diff'erent manufacturers 
(Symmetricom for CERN, ESAT for LNGS). Dedicated tests 
were performed to check for relative off'sets and time stability 
of the two systems |8]. The two systems are able to track each 
other within ±23ns over long periods. 

4. Conclusion 

The OPERA global readout system is the first fully Ethernet- 
based data acquisition system using the "smart sensors" con- 
cept. Commissioned during the first CNGS run in summer 
2006, it is running continuously with a high availability and ef- 
ficiency. During the 2008 physics run, around 10,000 neutrino 
interactions have been recorded in time coincidence with the 
CNGS, among which 1,700 interactions in the target. Emul- 
sions analysis is being performed by automatic scanning sta- 
tions ||9|]. 
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